Papers with word representation
Dive into Deep Learning for Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | GluonNLP is a powerful new toolkit that automates the most laborious aspects of deep learning for NLP. |
| Approach: | This hands-on tutorial demonstrates how to scale unsupervised pre-training techniques with Apache MXNet and GluonNLP. |
| Outcome: | This hands-on tutorial examines the challenges of scaling these models and algorithms effectively with Apache MXNet and GluonNLP. |
A Self-Supervised Integration Method of Pretrained Language Models and Word Definitions (2023.findings-acl)
Copied to clipboard
| Challenge: | DefBERT integrates word definition modeling with word semantics in dictionaries. |
| Approach: | They propose a method that integrates pretrained models with word semantics in dictionaries. |
| Outcome: | The proposed method is effective on word similarity and short text classification tasks. |
Exploiting WordNet Synset and Hypernym Representations for Answer Selection (2020.aacl-main)
Copied to clipboard
| Challenge: | Answer selection (AS) is a challenging subtask of document-based question answering (DQA). |
| Approach: | They propose to use WordNet to enrich the word representation and sentence encoding to incorporate similarity scores of two concepts that share synset or hypernym relations into the attention mechanism. |
| Outcome: | The proposed model outperforms existing state-of-the-art models on the public WikiQA and SelQA datasets and significantly improves the baseline system. |
Exploring Statistical and Neural Models for Noun Ellipsis Detection and Resolution in English (2020.aacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to noun ellipsis resolution have been sparse, using syntactic feature constraints for marking licensors and selecting their antecedents. |
| Approach: | They propose to use supervised machine learning to improve the existing F1 score by 16.55% and resolution by 14.97% for noun ellipsis subtasks. |
| Outcome: | The proposed framework improves the existing F1 score by 16.55% and the resolution subtask by 14.97%. |
Denoising Word Embeddings by Averaging in a Shared Space (2021.starsem-1)
Copied to clipboard
| Challenge: | Continuous word embeddings have been introduced several years ago as a standard building block for NLP tasks. |
| Approach: | They propose a method of fusing word embeddings that were trained on the same corpus but with different initializations. |
| Outcome: | The proposed method improves word embeddings on a range of tasks. |
A Flexible and Easy-to-use Semantic Role Labeling Framework for Different Languages (C18-2)
Copied to clipboard
| Challenge: | DAMESRL is an open source framework for deep semantic role labeling . language-specific characteristics and the available amount of training data influence the optimal model structure . |
| Approach: | They propose an open-source framework for deep semantic role labeling that is available under the Apache 2.0 license. |
| Outcome: | The proposed framework is available under the Apache 2.0 license. |
Can Network Embedding of Distributional Thesaurus Be Combined with Word Vectors for Better Representation? (N18-1)
Copied to clipboard
| Challenge: | Distributed representations of words learned from text have proved to be successful in various natural language processing tasks. |
| Approach: | They propose to embed a distributional thesaurus network into dense word vectors and compare them to state-of-the-art word representations. |
| Outcome: | The proposed representations improve performance against state-of-the-art word representations even without handcrafted lexical resources. |
Distributional Inclusion Vector Embedding for Unsupervised Hypernymy Detection (N18-1)
Copied to clipboard
| Challenge: | Existing unsupervised methods for learning hypernyms from unlabeled text are not scaled to large vocabularies or yield unacceptably poor accuracy. |
| Approach: | They propose an unsupervised method of hypernym discovery using word contexts . they use word2vec to embed word context distributions without supervision . |
| Outcome: | The proposed method provides double the precision and highest average performance on 11 datasets. |
Deep Generative Model for Joint Alignment and Word Representation (N18-1)
Copied to clipboard
| Challenge: | EmbedAlign model embeds words in their complete observed context and learns by marginalisation of latent lexical alignments. |
| Approach: | They exploit translation as a distributional context and embed words as posterior probability densities, rather than point estimates, which allows them to compare words in context using a measure of overlap between distributions. |
| Outcome: | The proposed model performs on a range of lexical semantics tasks and achieves competitive results on benchmarks including natural language inference, paraphrasing, and text similarity. |
From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Current models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. |
| Approach: | They propose a hierarchical open-vocabulary language model that adopts a shallow Transformer architecture to learn word representations from their characters and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. |
| Outcome: | The proposed model outperforms baselines on various downstream tasks and is robust to textual corruption and domain shift. |
A Comprehensive Comparison of Word Embeddings in Event & Entity Coreference Resolution. (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for Coreference Resolution rely on word embeddings for word representation, but performance of different embeddables is largely overlooked. |
| Approach: | They frame their study in the context of Event and Entity Coreference Resolution (EvCR & EnCR) they examine whether there is a trade-off between performance and embedding size . |
| Outcome: | The embeddings achieve 86% of the performance of the largest model while being 1.2% of its size. |
Imputing Out-of-Vocabulary Embeddings with LOVE Makes LanguageModels Robust with Little Cost (2022.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art NLP systems are brittle when faced with Out-ofVocabulary words . we present a framework that extends word embeddings and makes them robust to OOV . |
| Approach: | They propose a framework that extends existing word embeddings and makes them robust to OOV. |
| Outcome: | The proposed model performs better on original datasets and corrupted variants than previous competitors. |
Delta Embedding Learning (P19-1)
Copied to clipboard
| Challenge: | Unsupervised word embeddings have limitations to the semantics of words and inadequate fine-tuning of embedded word can lead to suboptimal performance. |
| Approach: | They propose a method that optimizes word embeddings by regularizing them incrementally to ensure they are tuned in an incremental way. |
| Outcome: | The proposed method improves performance on various NLP tasks and shows that it absorbs semantic information without "forging" |
Context Analysis for Pre-trained Masked Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models that learn contextualized word representations from a large un-annotated corpus have become a standard component for many downstream NLP tasks. |
| Approach: | They propose to use a masking and gradient approach to evaluate the impact of context on the word representation. |
| Outcome: | The proposed model architectures are architecture agnostic and gradient based. |
Exploring Sequence-to-Sequence Learning in Aspect Term Extraction (P19-1)
Copied to clipboard
| Challenge: | Aspect term extraction (ATE) aims at identifying all aspect terms in a sentence . sequence labeling based methods cannot make full use of overall meaning of sentence if they have dependencies between labels. |
| Approach: | They propose to formalize ATE as a sequence-to-sequence (Seq2Seque) learning task . they propose gated unit networks and position-aware attention mechanism to make it suit to ATE . |
| Outcome: | The proposed learning task is effective when labels correspond to words one by one . the proposed learning system is gated unit networks and position-aware attention mechanism . |
Time-Aware Word Embeddings for Three Lebanese News Archives (2020.lrec-1)
Copied to clipboard
| Challenge: | a large corpus of newspaper archives has been generated, but historians have struggled to analyze it manually for decades. |
| Approach: | They propose to train word embeddings from three large Lebanese news archives, which collectively consist of 609,386 scanned newspaper images and span 151 years. |
| Outcome: | The embeddings are trained using a Google Tesseract 4.0 OCR engine and a benchmark of analogy tasks to evaluate their accuracy. |
MICRON: Multigranular Interaction for Contextualizing RepresentatiON in Non-factoid Question Answering (D19-1)
Copied to clipboard
| Challenge: | Existing approaches for non-factoid question answering can be categorized into representation and interaction focused approaches. |
| Approach: | They propose a novel approach which derives contextualized uni-gram representation from n-grams. |
| Outcome: | The proposed approach achieves state-of-the-art in two public non-factoid question answering datasets. |
Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit? (2020.lrec-1)
Copied to clipboard
| Challenge: | In recent years, vectors derived from neural network training have replaced count-based distributional semantic models as a de facto standard for word representation in NLP. |
| Approach: | They propose to evaluate count models and word embeddings on thematic fit estimation by taking into account a larger number of parameters and verb roles and introducing dependency-based embedders in the comparison. |
| Outcome: | The proposed model outperforms count models and word embeddings in thematic fit estimation tasks while introducing dependency-based embedders. |
Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models often encode word-form variation as linear directions in the embedding space. |
| Approach: | They propose a compact reshaping of large language models' vocabulary by using shared vectors instead of unique tokens. |
| Outcome: | The proposed approach frees 10-40% of vocabulary slots to be reallocated where tokenization is inefficient. |